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1.  Introduction 


The  present  analysis  deals  with  very  high-dimensional  data  sets, 
each  one  containing  close  to  nine  hundred  binary  variables.  Each  data 
set  corresponds  to  an  evaluation  of  one  complex  system.  These  data  sets 
are  characterized  by  large  portions  of  missing  data  where,  moreover,  the 
unobserved  variables  are  not  the  same  in  different  evaluations.  Thus, 
the  problems  which  confront  the  statistical  analysis  are  those  of-  multi¬ 
variate  binary  data  analysis,  where  the  number  of  variables  is  much 
larger  than  the  sample  size  and  in  which  missing  data  varies  with  the 
sample  elements.  The  variables,  however,  are  hierarchically  structured 
and  the  problem  of  clustering  variables  to  groups  does  not  exist  in  the 
present  study.  In  order  to  motivate  the  statistical  problem  under 
consideration,  the  Marine  Corps  Combat  Readiness  Evaluation  System 
(MCCRES)  is  described  for  infantry  battalions  and  then  used  for  exposi¬ 
tion.  The  present  paper  provides  a  statistical  model  for  data  from  MCCRES 
and  develops  estimation  and  prediction  procedures  which  utilize  the 
dependence  structure.  The  E-M  algorithm  is  applied  to  obtain  maximum- 
likelihood  estimates  of  the  parameters  of  interest.  Numerical  examples 
illustrate  the  proposed  methods. 


THE  GEORGE  WASHINGTON  UNIVERSITY 
School  of  Engineering  and  Applied  Science 
Washington,  DC  20052 

Institute  for  Management  Science  and  Engineering 


STATISTICAL  ANALYSIS  OF  VERY  HIGH-DIMENSIONAL  DATA  SETS 
OF  HIERARCHICALLY  STRUCTURED  BINARY  VARIABLES  WITH  MISSING  DATA 
AND  APPLICATION  TO  MARINE  CORPS  READINESS  EVALUATIONS 


by 

S .  Zacks 
W.  H.  Marlow 
S.  S.  Brier 


Abstract 

of 

READINESS  RESEARCH 
GWU/IMSE/ Serial  T-481/83 
31  December  1983 


The  present  analysis  deals  with  very  high-dimensional  data  sets, 
each  one  containing  close  to  nine  hundred  binary  variables.  Each  data 
set  corresponds  to  an  evaluation  of  one  complex  system.  These  data  sets 
are  characterized  by  large  portions  of  missing  data  where,  moreover,  the 
unobserved  variables  are  not  the  same  in  different  evaluations.  Thus, 
the  problems  which  confront  the  statistical  analysis  are  those  of  multi¬ 
variate  binary  data  analysis,  where  the  number  of  variables  is  much 
larger  than  the  sample  size  and  in  which  missing  data  varies  with  the 
sample  elements.  The  variables,  however,  are  hierarchically  structured 
and  the  problem  of  clustering  variables  to  groups  does  not  exist  in  the 
present  study.  In  order  to  motivate  the  statistical  problem  under 
consideration,  the  Marine  Corps  Combat  Readiness  Evaluation  System 
(MCCRES)  is  described  for  infantry  battalions  and  then  used  for  exposi¬ 
tion.  The  present  paper  provides  a  statistical  model  for  data  from  MCCRES 
and  develops  estimation  and  prediction  procedures  which  utilize  the 
dependence  structure.  The  E-M  algorithm  is  applied  to  obtain  maximum- 
likelihood  estimates  of  the  parameters  of  interest.  Numerical  examples 
illustrate  the  proposed  methods. 


Research  Supported 
by 

Contract  N00014-83-K-0216 
Project  NR  347-131 
Office  of  Naval  Research 


T-481 


The  Marine  Corps  Combat  Readiness  Evaluation  System  measures  the 
effectiveness  of  Marine  Corps  units,  including  infantry  units  (battalions), 
against  well  defined  Mission  Performance  Standards  (MPS)  in  simulated 
combat  environments.  The  evaluation  system  for  infantry  battalions  is 
comprised  of  17  MPSfs  which  are  classified  into  four  sections.  Section  A 
contains  three  MPSfs  which  are  applicable  to  all  evaluations.  Section  B 
contains  seven  MPSfs  pertaining  to  amphibious  assaults  and  normal  combat 
operations.  Section  C  has  four  MPSfs  on  specialized  combat  operations 
and  Section  D  has  three  which  deal  with  use  of  outside  support  assets. 

In  all  cases,  each  MPS  consists  of  several  tasks  and  each  task  consists 
of  several  requirements. 

In  the  present  paper  the  term  evaluation  refers  to  the  process 
of  evaluating  the  readiness  of  an  infantry  battalion.  In  an  evaluation 
a  team  of  judges  assigns  the  combat  unit  a  value  of  1  or  0  on  each 
requirement  being  evaluated,  according  to  whether  the  performance  for 
that  requirement  is  satisfactory  or  not  and  an  N/A  if  the  requirement 
was  not  applicable  (was  not  evaluated) .  A  weighted  average  of  all  the 
scores  for  the  applicable  requirements  in  a  given  task  is  then  determined. 
For  example,  if  in  a  given  task  there  are  6  requirements,  and  the  given 
data  are 


requirement 

1 

2 

3 

4 

5 

6 

score 

0 

1 

1 

N/A 

N/A 

1 

weight 

.25 

.083 

.083 

.083 

.25 

.25 

The  weighted  applicable  task  score  is 


0  x  .25  +  1  x  .083  -fix  .083  +  1  x  ,25 

.25  +  .083  +  .083  +  .25 


-  2  - 


.625 
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The  weights  for  these  requirements  have  been  determined  for  the 
MCCRES  by  teams  of  experts.  Accordingly,  each  applicable  task  is 
represented  by  a  weighted  average,  ranging  between  zero  and  one.  This 
weighted  average  will  be  called  the  observed  task-score .  The  present 
MCCRES  system  computes  hierarchically  a  weighted  average  of  all  the  task- 
scores  within  an  MPS,  to  obtain  an  MPS-score  for  each  unit;  a  weighted 
average  of  all  the  MPS-scores  to  obtain  a  section-score  and  a  weighted 
average  of  all  section-scores  to  obtain  a  summarizing  total-evaluations- 
score  (TES) .  These  weighted  averages  are  computed  as  in  the  case  of  the 
observed  task  scores.  If  all  the  requirements  in  a  given  task  are  N/A, 
the  task  itself  is  called  non-applicable  (N/A).  Similarly,  if  all  tasks 
in  a  given  MPS  are  N/A  the  MPS  is  called  N/A.  The  weighted  averages  of 
the  task  scores  in  an  MPS  are  based  only  on  the  applicable  tasks. 

Similarly,  the  weighted  averages  of  MPS  scores  in  the  four  sections  are 
based  only  on  the  applicable  MPS  scores.  Thus,  every  unit  can  be 
characterized  after  an  evaluation  by  a  profile  consisting  of  seventeen 
vectors  of  task-scores,  one  for  each  MPS;  by  four  vectors  of  MPS-scores, 
one  for  each  section;  by  a  vector  of  four  section-scores  and  a  TES.  In 
Table  1.1  we  provide  an  example  of  such  a  profile  of  scores  for  evaluation 
No.  1.  Other  evaluations  have  the  same  hierarchical  scores  structure  but 
the  non-applicable  tasks  or  MPS’s  might  vary  from  one  evaluation  to 
another.  This  lack  of  uniformity  in  the  applicable  and  non-applicable 
requirements  from  one  evaluation  to  another  creates  severe  data  deficiency. 
The  observed  task  scores  may  not  reflect  correctly  the  ability  of  a  unit 
(battalion)  to  perform  a  given  task.  The  task  scores  depend  heavily  on 
the  design  of  the  evaluation  in  terms  of  which  requirements  are  non- 
applicable.  This  can  be  illustrated  by  using  the  previous  example.  If 
requirements  4  and  5  were  applicable  and  the  scores  obtained  on  these 
requirements  are  1  and  0,  respectively,  the  task  score  would  have  been 
0  x  .25  +  1  x  .083  +  1  x  .083  +  1  x  .083  +  0  x  .25  +  1  x  .25  =  .50  . 
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Table  1.1 


Hierarchical  Scores  of  A  Particular  Evaluation 


*) 


Level 


TES  .800 


Section 

A 

B 

C 

D 

.745 

.843 

.863 

.930 

MPS 

1 

2 

3 

4 

2. A. 

.633 

.928 

.755 

2.B. 

.935 

.718 

.854 

.850 

2.C. 

.XXX 

.772 

.XXX 

1.000 

2.D. 

1.000 

.841 

.889 

TASK 

1 

2 

3 

4 

2.A.I. 

.757 

.568 

.848 

.575 

2. A. 2. 

.940 

1.000 

1.000 

.855 

2. A. 3. 

.742 

.770 

1.000 

1.000 

2.B.I. 

.962 

1.000 

1.000 

.733 

2.B.2. 

1.000 

1.000 

1.000 

.536 

2.B.3. 

1.000 

.960 

1.000 

.680 

2.B.4. 

.827 

.680 

.743 

.625 

2.B.5. 

.XXX 

.XXX 

.XXX 

.XXX 

2.B.6. 

.797 

.891 

.800 

.XXX 

2.B.7. 

1.000 

.727 

.XXX 

.XXX 

2.C.I. 

.XXX 

.XXX 

.XXX 

.XXX 

2.C.2. 

1.000 

1.000 

.543 

2.C.3. 

.XXX 

.XXX 

.XXX 

.XXX 

2.C.4. 

1.000 

1.000 

1.000 

2.D.I. 

1.000 

1.000 

1.000 

1.000 

2.D.2. 

2.D.3. 

.851 

.888 

1.000 

.890 

.316 

1.000 

5 

6 

7 

.XXX 

.843 

.849 

5 

6  7  8 

9 

10  11 

.500 

.550  .331  .857 

.593 

.429  .833 

1.000 

1.000  .783  .857 

.852 

1.000  1.000 

1.000 

.667  .603  .239 

1.000 

.500  .652 

.865 

1.000 

.839  1.000  1.000 

.XXX 

.XXX 

•XXX  .XXX 

.XXX 

.XXX 

.862 

.XXX 

.XXX 

.XXX 

.XXX 

1.000 

1.000  1.000  1.000 

1.000 

*) 


.XXX 


represents  a  non-applicable  task  or 


a  non-applicable  MPS . 


12 

.647 

.937 
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The  previous  score  of  .625  is  overestimating  the  performance  of  the 
task  by  25%.  The  problem  is  therefore  to  develop  a  method  of  assessing 
the  statistical  characteristics  of  the  task  scores  in  the  population 
of  all  possible  evaluations,  and  suggest  a  method  of  adjusting  the 
individual  task  scores,  in  cases  of  N/A  requirements,  to  better 
estimate  the  capability  of  the  unit.  The  present  paper  addresses 
this  problem  of  developing  proper  estimation  procedures  of  the  basic 
structural  parameters,  so  that  meaningful  prediction  of  the  missing 
values  and  adjustment  of  the  task  scores  will  be  possible.  The  analysis 
is  based  on  a  data  set  of  75  evaluations.  These  evaluations  were  conducted 
on  different  combat  units,  some  of  which  were  evaluated  only  partially. 

In  particular,  Sections  B,  C  and  D  of  the  evaluations  have  a  large 
number  of  N/A  requirements  in  many  evaluations.  We  therefore  restrict 
attention  in  the  present  paper  to  Section  A,  which  consists  of  3  MPSfs 
encompassing  31  tasks.  As  mentioned  earlier,  the  statistical  problem 
discussed  in  the  present  paper  is  that  of  analyzing  multivariate  binary 
datasetshaving  an  extensive  amount  of  incomplete  or  missing  data.  As 
such,  the  interest  in  the  methods  developed  here  is  beyond  the  particular 
scope  of  MCCRES  evaluations.  The  results  of  the  present  study  may  well 
be  applicable  to  the  fields  of  reliability  of  complex  systems,  signal 
processing,  etc. 
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2.  The  Data  Structure  and  Basic  Definitions 

In  the  present  section  we  discuss  the  structure  of  the  data  file, 
on  which  the  analysis  of  the  present  study  is  based,  and  provide  some 
basic  definitions.  As  mentioned  earlier,  we  concentrate  on  Section  2. A. 

The  discussion  is  therefore  confined  to  the  structure  of  the  data  in  this 
section.  The  data  related  to  the  other  three  sections  is  structured 
similarly.  As  seen  in  Table  1.1,  Section  A  is  comprised  of  three  MPSTs, 
consisting  of  12,  12  and  7  tasks,  respectively.  Each  of  these  31  tasks 
is  structured  around  a  certain  number  of  requirements.  The  number  of 
requirements  may  differ  from  one  task  to  another.  Moreover,  not  all  the 
units  were  evaluated  on  the  same  requirements.  Thus,  if  a  unit  was  not 
evaluated  on  a  given  requirement,  that  requirement  is  non-applicable 
for  that  unit.  In  Table  2.1  we  present  data  on  requirements  of  two 
particular  evaluations.  The  value  1  signifies  that  the  requirement  was 
performed  satisfactorily.  The  value  0  was  assigned  if  the  performance 
was  unsatisfactory  and  the  value  9  designates  a  non-applicable  require¬ 
ment.  We  see  in  Table  2.1  that  the  number  of  requirements  for  the  31 
tasks  of  Section  2. A  are,  respectively,  9,  5,  5,  5,  9,  10,  10,  8,  10,  7, 

6,  6,  7,  4,  7,  8,  9,  8,  7,  10,  9,  8,  6,  5,  11,  7,  6,  7,  9,  9  and  7. 

We  also  see  that  in  EVAL  1  only  requirement  2.A.2.11.5  was  not  applicable 
while  in  EVAL  2  there  are  several  such  requirements.  In  addition,  there 
are  differences  due  to  the  performance  level  of  the  two  units.  We  define 
below  the  pertinent  variables  for  the  structure  of  the  data  in  the  present 
study. 

Let  i  denote  the  index  of  an  MPS  within  Section  2. A,  i  =  1,  2,  3  . 
Let  t^(i  =  1,  2,  3)  denote  the  number  of  tasks  in  the  i-th  MPS  of  2. A, 

and  let  j,  j  =  l,...,t^  ,  denote  the  index  of  the  j-th  task  of  MPS  2.A.i. 

Let  r (i  =  1,...,3;  j  =  l,...,t^)  denote  the  number  of  requirements 

in  task  2.A.i.j  and  let  k  ,  k  =  l,...,r  ,  be  the  index  of  the  k-th 

requirement  within  2.A.i.j  .  Finally,  let  n  ,  n  =  1,...,N  (N  =  75)  , 
be  the  index  of  the  n-th  evaluation  (unit).  We  introduce  two  basic 
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Table  2.1 

Requirement  Data  For  Evaluations  1  and  2 


Task 

2.A.1.01  1 
2. A. 1.02  0 
2.A.1.03  1 
2. A. 1.04  0 
2.A.1.05  1 
2.A.1.06  1 
2. A. 1.07  1 
2. A. 1.08  1 
2. A. 1.09  1 
2.A.1.10  1 
2. A. 1.11  1 
2.A.1.12  1 
2. A. 2. 01  0 
2. A. 2.02  1 
2. A. 2. 03  1 
2.A.2.04  1 
2. A. 2. 05  1 
2. A. 2.06  1 
2. A. 2. 07  1 
2.A.2.08  1 
2. A. 2.09  1 
2. A. 2. 10  1 
2. A. 2. 11  1 
2. A. 2.12  0 
2.A.3.01  1 
2. A. 3.02  1 
2.A.3.03  1 
2. A. 3.04  1 
2.A.3.05  1 
2.A.3.06  0 
2.A.3.07  0 


EVAL  1 
Req. 

0  10  11 
1110 
0  111 
0  111 
0  0  110 
0  10  10 
0  0  0  0  1 
10  111 
110  0  1 
110  0  0 
110  11 
10  0  11 
11111 
111 
11111 
110  11 
11111 
11111 
10  111 
1110  1 
11110 
11111 
1119  1 

1111 
110  11 
110  0  1 
11111 
11111 
10  110 
1110  0 
110  0  0 


111 


Oil 
10  11 
0  0  11 

1  1 

0  0  11 

0 


1 

1 

1  1 
111 
1  1 
1 

1111 
111 
1  1 


0  0  111 
1 

1 

110 

111 

0 


Task 

2. A. 1.01  1 
2. A. 1.02  1 
2. A. 1.03  1 
2. A. 1.04  0 
2. A. 1.05  1 
2. A. 1.06  1 
2. A. 1.07  1 
2. A. 1.08  1 
2. A. 1.09  0 
2.A.1.10  0 
2. A. 1.11  1 
2. A. 1.12  1 
2. A. 2. 01  1 
2. A. 2. 02  1 
2. A. 2. 03  1 
2. A. 2. 04  1 
2. A. 2. 05  1 
2. A. 2.06  1 
2. A. 2.07  1 
2.A.2.08  1 
2.A.2.09  1 
2.A.2.10  1 
2. A. 2. 11  0 
2. A. 2. 12  1 
2. A. 3. 01  1 
2. A. 3.02  1 
2. A. 3.03  1 
2. A. 3.04  1 
2. A. 3.05  1 
2. A. 3. 06  1 
2. A. 3.07  1 


EVAL  2 

Req . 

0  10  10 
1111 
110  1 
0  0  0  1 

11111 
11111 
11111 
1110  0 
0  110  0 

0  0  9  9  9 

11111 
10  0  11 

11111 
111 
11111 
11111 
11111 
11111 
11111 
11110 
11111 
11111 
0  19  11 

1111 
11111 
11111 
11111 
110  11 
11111 
110  0  0 
10  110 


110 


Oil 
1111 
1111 
1  1 

1111 

9 


1 

1 

9  9 

111 
1  1 
1 

1111 
111 
1  1 


11111 

1 

1 

110 
0  10 
1 
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variables:  an  incidental  variable 


(2.1). 


1  ,  if  requirement  2.A.i.j.k 

is  applicable  for  the  n-th  evaluation 

0  ,  otherwise 


and  a  random  variable 


(2.2) 


1  ,  if  requirement  2.A.i.j.k  is 

satisfied  at  the  n-th  evaluation 

0  ,  otherwise 


The  incidental  variables  are  design  dependent,  while  the  random 


variables  follow  some  probabilistic  model  which  is  specified  in 


Jijk 


Section  3. 


Let  w 


ijk  denote  the  weight  assigned  to  requirement  2.A.i.j.k  . 


r.  . 
ij 


These  are  non-negative  values  such  that  E 

k=l 


Wijk  =  1  f°r  a11 


Let 

(2.3) 


W, 


(n) 


ij 


r?  <ni  »■ 

k=1  ijk  xjk 


be  the  sum  of  applicable  weights  for  task  2.A.i.j  at  the  n-th  evaluations. 
Accordingly,  the  task-score  of  the  n-th  evaluation  is  the  random 

variable 


(2.4) 


x<n) 

IJ 


J^l  w  /W 
k=1  ijk  ijk  ijk  ij 


If  at  the  n-th 
not  applicable 

for  all  k=l, . 


evaluation  all  the  r_  requirements  of  task  2.A.i.j 

then  wty  =  0  and  we  define  =  N/A  .  If 

iJ  ij  ijk 

•  we  say  that  the  task  score  is  complete. 


are 
=  1 
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If  some  requirements  are  non- applicable  the  corresponding  task  score  is 
incomplete. 


3.  The  Statistical  Model  and  the  Basic  Parameters 


Let 


i  (n) 

ijk 


be  the  probability  that  requirement 


i- j  .k 


is  satisfied 


at  the  n-th  evaluation.  These  values  vary  from  one  evaluation  to  another 
according  to  some  distribution  of  these  parameters.  More  specifically, 
we  assume  that. 


(3.1) 


j -  i| 

ljk  1  ljk 


I  =9S 


we  further  assume  that  j  5  n=l,2,...,Nj  are  conditionally  independent. 


given  {6^  5  n=l,...,N}  and  that  {0^  ;  n=l,...,N>  are  independent 

random  variables,  having  a  common  (unknown)  distribution. 

Let  n.jk  -  E  «<">>  .  Thus,  P  {J<">  -  1}  -  H.  ,k  for  all  n-1 . N  . 

Furthermore,  {J^?k  ;  n=l,...,N}  can  be  considered  as  a  sequence  of 

independent  random  variables  representing  N  Bernoulli  trials.  Thus, 
n±jk  is  the  probability  that  a  randomly  selected  evaluation  has  satisfied 

requirement  i.j.k.  The  expected  complete  task  score  (ECTS)  for  a  randomly 
selected  evaluation  is 


,(n) 


(3.2) 


r .  . 

ij 

x . .  =  z  w.  n . 

1J  k=i  ljk  ljk 


Other  parameters  of  interest  of  the  (super)  population  of  all  possible 
evaluations  are: 


/  \ 

(i)  Variances  and  covariances  of  J..;  ; 

ijk 

(ii)  Variances  and  covariances  of  CTSfs. 

(iii)  Correlations  among  CTSfs. 
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The  covariances  of  and  Jijk^  *  within  a  randomly  chosen  evaluation 

are: 


(3.3)  cov|j<”>  ,  )W]. 


,  k  =  k' 


p.,(k,kO  -  n. ..  n. .. k  +  r 

ij  ljk  ljk  ' 


where 


(3.4)  P.,(k,k")  =  P  |  jfn>  jf“>  =  lj 

ij  l  ljk  lj  k  > 


The  covariances  given  by  (3.3)  will  be  denoted  by  0  (k,k')  ,  and  the 

covariance  matrix  of  requirements  within  the  i.j  task  is  denoted  by  (V. .) 

(Vi-j)  is  a  positive  definite  matrix  for  every  i.j  .  The  variance  of  a 
J  r. 


% 


(n)  . 


ij 


,(n) 


complete  task  score,  X)“'  =  Z  w  J ,  is  given  by 

13  13  k  ljk 


(3.5) 


o1  =  V  {  xfn) J  =  w:.  (V. .)  w. .  , 
a3  (  13  )  0,13  13  'vAl 


where  w  -  (w  . w. .  )  is  the  vector  of  weights  of  requirements 

rl»-LJ  -‘■J  j 

within  the  (i.j)th  task.  The  covariance  of  two  different  complete  task 
^(n)  ^(n) 

scores,  and  X_^  >.  ,  within  the  i-th  MPS  can  be  written  as 


(3.6) 


cov(X^n)  ,  xfnb  =  wr.  (K.  .  ..)  w... 
13  13  ^13  i'33  ^13 


where  (K .  ..>•)  is  an  r.  .  x  r..^  matrix  of  covariances  C.  .  .<.(k,£)  = 
1  JJ  !3  13  i*33 


Extending  the  previous  definition,  we  have 


(3.7) 


ci.jr(k,«  -  -  n.Jk  nirt  , 
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where  P.  .  ..(k,£)  =  P  fjf^  =  lj  .  Let  C.(j,j")  denote  the 

i*jj  (  ljk  lj  Z  f  1  J’J 

covariances  defined  in  (3.6).  The  correlations  between  two  complete 
task  scores  in  the  same  MPS  are  then 


(3.8) 


p.  =  C.(j,jO/  cr.  .  a.  .  ^  ,  j,j^  =  l,...,t. 
i-JJ  iVJ  ,J  lj  it  *  J  ,J  *  ’  i 


ij  iJ 


In  the  following  section  we  present  estimators  of  these  parameters  and 
discuss  their  properties. 

4.  Estimating  the  Basic  Parameters 
(N) 

Let  denote  the  number  of  evaluations,  among  the  N  given 

ones,  in  which  requirement  i.j.k  is  applicable,  i.e., 


m<n>  =  z 

ljk  n=1  13k 


We  assume  that 


and 


(i)  >  0  *  for  aH  i.j.k  , 


(ii)  -*  oo  as  N  ,  for  all  i.j.k. 

ljk  -  J 


4.1  Estimating  ECTS 


The  following  is  an  unbiased  and  strongly  consistent  estimator 


of  nijk  ;  namely> 


(4.1) 


=  z  z^l 

ljk  ,  ljk  ljk  ljk 


(  ~ (N)  ) 

Indeed,  according  to  assumption  (i) ,  =  anc*»  according 

to  assumption  (ii)  and  the  strong  law  of  large  numbers  (SLLN) 
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'UN)  a,S‘ 


n,v*.Y  -*■  II.  M  as  N  -»■  00  .  Moreover,  since  for  each  i.i.k  ,  J are 

1JK  ljk  Ijk 

independent  and  identically  distributed  (IID) ,  n=l,...,N  ,  the  distribution 
N 

°f  Sijk  =  J/ijk  Jijk  ±S  the  binomia1’  >  n..k)  .  Thus,  the 

~(N) 

estimator  is  asymptotically  normal  with  mean  IL ^  and  variance 

(N) 

O.  .  (k,k)/M_k  ,  where  a_(k,k)  =  IL^  (1  "  .  Thus,  confidence 


ij 


intervals  for  IL  ^  can  determined  in  the  usual  manner 
(see  Zacks  [1981,  pp.  268]), 

An  unbiased  and  strongly  consistent  estimator  of  the  ECTS  ,  x^  ,  is 


(4.2) 


iJ  k=1  ijk  ijk 


/s  (N) 

Indeed,  x_  is  a  linear  combination  of  unbiased  and  strongly  consistent 

^  (N) 

estimators.  The  variance  of  x ; .  is  given  by 


(4.3) 


vlx^}  =  wr.  (d<nV--  . 

(  )  'XAJ  \  1J  /'Uij 


(N) 

where  (D^  )  is  a  positive  definite  symmetric  matrix  whose  elements  are 


(4.4) 


D<“><k,k') 

1J 


m<n>  (k.O 

a  (k  k')  _ 

aij(k’k  }  M(N)  (H) 

ijk  Mijk" 


k,k"=i. 


’rij  ; 


where 


(4.5) 


M^N)  (k,k")  =  Z  Z^n)  Z 
ij  n=1  ijk  ijk 


Notice  that  D^^(k,kO  depends  on  the  zf1?^  values.  We  assume  that 
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M^^(k,k^)  >  0  for  all  i.j  and  all  k,k>=l, . . . ,r . .  .  Moreover  ,  we  assume 
^■Jfc  ij 

(iii)  (kjk'')  -*■  00  as  N  ■+  00  ,  all  i.j  ,  k,k"'=l, . . . ,r. .  . 

ij  ij 

If  =  1  f°r  aH  i«j«k  and  all  n=l,...,N,  then  D^^(k,k^)  = 

a^j(k,k^)/N  .  This  is  generally  not  the  case!  It  is  simple  to  show  that 


(4.6) 


M^}(k,kO  , 

- ^ -  <  - 


Thus,  if  =  0(N) 


then  V  I  =  0(N  b  .  In  Section  5  we  will 


develop  estimators  of  II..,  and  of  t  ,  .  ,  which  utilize  the  covariances 

ijk  ij 

between  requirements. 

4.2  Estimating  Variances  and  Covariances 

We  start  with  the  development  of  estimators  of  the  variances  and 
covariances  of  the  requirement  variables  ,  i.e.  (k,kO  . 

A  strongly  consistent  estimator  of  (k,k^)  is 


(4.7) 

where 

(4.8) 

g(N) 

ij 

Indeed,  since 

m<n)( 

3<« (k,k' >  -  p<«  (k.kO  -  ii<N>  n<« 

ij  ij  ijk  ljk 


N 


z  Z ,  jty  jf“b 


M^N)(k,kO  n=l  ijk  ijk"  ±jk  ijk 
IJ 


IJ 


i(N), 

ij 


a.s . 


ij 


-(N) 


a.s, 


as  N  00  by  the  SLLN.  Hence  a.  .  (kjk-*)  -*  a..(k,kO  as  N  -* 

ij  ij  , 
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Thus,  if  the  values  of  M^^(k,k^)  are  large  for  all  k  and  k'  , 

the  estimates  of  a_(k,k^)  will  be  close  to  the  true  values,  with  high 

probability.  If,  however,  some  of  the  values  of  M^^(k,kO  are  not 

ij  k 

sufficiently  large  some  algebraic  inconsistencies  might  occur  in  the  values 

_  ^ 

of  the  estimators  since  due  to  the  missing  values,  II../  ,  II.. /^  and 

i  j  k.  ijk 

^ (k,k  )  are  not  necessarily  based  on  the  same  sample  values.  The 

following  is  an  illustration  of  this  point  in  a  case  of  an  extremely 
small  sample.  Suppose  that  we  have  three  requirements  with  the  following 
data. 


n 

req . 

12  3 

i 

19  0 

2 

0  19 

3 

9  0  9 

4 

111 

5 

19  1 

6 

9  11 

The  statistics  of  this  small  sample  are: 


n  =  6  ,  m^6)  =  m^6)  =  m<6)  =  4,  =  n3 


(6)  _  jjj-6)  =  n(6) 


f(6) 

12 


m;:7  =  2  ,  m;~'  =  3,  m 


i(6)  = 
13 


(6) 

23 


=  2  ,  =  .50  ,  P 


(6) 

13 


,(6)_ 


^23  =  ^  *  ^he  corresPonding  estimator  of  (V)  is 


.75  , 
.67  , 


.1875 

-.0625 

.1075 

-.0625 

.1875 

.4375 

.1075 

.4375 

.1875 
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This  matrix  is,  however,  not  positive  definite  as  a( 2,3)  >  8(2,2)  8(3,3)  . 
This  result  is  obviously  inadmissible!  Such  algebraic  inconsistencies, 
when  covariances  are  estimated  from  incomplete  data  sets,  have  been 
previously  mentioned  in  the  literature  (see  Little  (1979)). 

In  the  following  section  we  derive  pair-wise  maximum  likelihood 
estimators  (PMLE)  of  P_(k,k")  and  of  P±mjy(k,Z)  .  These  PMLE  yield 

algebraically  consistent  estimators  of  the  covariances  between  any  pair 

of  requirements  and  also  more  efficient  estimators  of  the  marginal 

probabilities  II.  ..  . 

ijk 

5.  Pairwise  Maximum  Likelihood  Estimators  (PMLE) 


Consider  any  two  requirements,  say  (i.j.k)  and  (i,j^.£)  .  We  are 
interested  in  estimating  C  ^(k,£)  =  P .  .  .^(k,£)  -  II.  II.. .  If 

1*JJ  1  JJ  1 J  K-  1J  A/ 

there  were  no  missing  values  in  the  data  set,  i.e.,  if  all  =  1  , 

ljk 

then  the  covariance  estimators  presented  in  Section  4.2  are  also  maximum 

likelihood  (MLE)  and  thus  best  asymptotically  normal  (BAN)  (see  Zacks 

[1981;  PP*  249])  •  This  is  not  the  case,  however  under  missing  value 

situations.  It  will  be  shown  that  since  different  requirements  are  not 

independent,  the  MLE  of  P.  ..^(k,£)  and  of  II...  ,  II..  ,  utilize 

i-jj  ljk  ij  l 

more  of  the  information  in  the  data  than  the  unbiased  estimators  discussed 
in  the  previous  section.  The  initial  step  in  deriving  the  MLETs  is  the 
explicit  (or  implicit)  formulation  of  the  likelihood  function  of  the 
parameters  given  the  data.  Due  to  the  missing  value  problem,  one  should 
formulate  these  likelihood  functions  in  terms  of  all  the 

Q  t. 

3  i 

II  II  2  ij  joint  probabilities  II(£ 

±=i  j=i  111  itirt1  JJ-L 

^3t  r  )  for  ^ijk  =  0  >  1  >  of  tlie  possible  realizations  of  the 
3  t3 


random  variable  J, ,J_ 

111  ^C3rt 


Since  we  have  only  N=75  evaluations. 


-  15 


T-481 


the  data  sets  will  have  zero  frequencies  for  most  of  the  corresponding 
cells.  We  therefore  resort  to  an  ad-hoc  solution,  in  which  we  estimate 

by  the  MLE  based  on  the  data  available  for  each  pair  of 

requirements  separately.  These  estimates  are  called  pairwise  maximum 
likelihood  (PMLE) .  The  PMLE  of  P„^(k,J2.)  coincides  with  the  MLE  of 

when  there  are  no  missing  values. 

5 . 1  Derivation  of.  the  PMLE  and  their  Asymptotic  Covariance  Matrix 

5.1.1  The  Likelihood  Equation 

Consider  any  two  requirements  (i.j.k)  and  (i.j'.il)  ,  within  the 
i-th  MPS.  For  the  purpose  of  simplifying  notation,  we  will  disregard 
in  the  present  section  the  actual  indexes  of  these  requirements,  and 
label  the  corresponding  variables  by  and  Z ^  , 

n=l,...,N  .  The  sufficient  statistic  for  the  joint  distributions  of 
these  variables  are  given  in  the  following  table. 

Table  5.1 

Frequency  Distribution  of  (Z1# J1)&(Z2> J2) 
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where 


00 


2  Z^n)  Z2(n)  (1-J^n))(l-J^n)) 
n=l 


01 


2  Z<n)  Z2(n)  (1-J^n))  J2n) 
n=l 


'10 


2  Z^n)  Z2(n)  jjn)(l-J^n)) 

n_i 


and 


=  _  y  7(n)  _(n)  T(n)  _(n) 

'll  "  Vl  z2  J1  J2 

n=l 


g0.  =  z  z<n)  (1-Z^n))  (1-J^n)) 

n=l 


2  Z^n)  (1-Z^n))  J^n) 
n=l 


2  (1-Z^n))  Z^n)  (1-J^n)) 

n=l 


I  (1-2 <n))  Z<n)  J<"> 

n_i 


2  (1-Z^n))  (1-Z^n)) 

n=l 


Let  M  =  fQ0  +  fQ1  +  f1Q  +  fu  .  Obviously, 

N  =  M  +  g^.  +  g.^  +  S.q  +  g,-^  +  h  •  In  case  of  complete  data  M  =  N 


The  frequencies  g.  (i=0,l)  and  g.  (i=0,l)  are  of  partially  observed 
1  •  JL 

data,  h  is  the  number  of  evaluations  in  which  both  requirements  are 
missing.  / 
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The  cell  probabilities  are  denoted  by 

a  =  P  j  J  <n)  =  0  ,  J^n)  =  0  }  , 

8  =  P  {  j{n)  =  0  ,  J^n)  =  1  J 

and 

Y  =  P  {  J^n)  =  1  ,  J<n)  =  0  }  . 

Accordingly,  the  kernel  of  the  log-likelihood  of  (ct,8,Y)  is 
(5-1>  Ka.B.Y)  =  fQ0  log  a  +  fQ1  log  8  +  f1Q  log  y 

+  fll  108  (1-a_B“Y)  +  80.  log  (a  +  8) 

+  8j_.  log  (l-a-8)  +  g<Q  log  (a  +  y) 

+  8.]_  log  (1-a-y)  . 


Indeed,  ^f  (£qq»  I-^q*  ^11^  ^as  a  mulblnomlal  distribution  while 

(8q.  j  gj_.)  anci  (g.Q  »  g.j)  ate  binomial,  independent  of  f  . 

'Xj 

5.1.2  Determination  of  FMLE  by  the  EM-Algorithm 

A  A  A  A 

The  vector  0  =  (a,  8,  y) ,  within  the  simplex  0  =  {0  :  0  <  a. 8  Y 

*\j  f\j  —  *  M  9  ' 

and  a  +  8  +  Y  <  1}  which  maximizes  (5.1)  is  called  the  PMLE  of 
0  =  (a,  8,  y)  .  Derivation  of  the  PMLE  can  be  obtained  either  by  solving 
the  set  of  nonlinear  equations 


(5.2) 


(V).£(0)  =  0  , 

y  >\j 
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or  by  applying  the  E-M  algorithm  of  Dempster,  Laird  and  Rubin  (1977). 
The  E— M  algorithm  requires  several  iterations,  but  the  computations  are 
simpler  than  those  required  to  solve  (5.2)  iteratively.  This  is  due  to 
the  fact  that,  in  the  case  of  N=M,  the  solution  of  (5.2)  within  0 
is  simply 


(5.3) 


a 


£00/N 


y 


6 


* 

f01/s 


y 


and  y  =  f *q/N  . 


where  f^  (i»j— 0,1)  designates  the  cell  frequencies  based  on  the 

complete  data.  Generally,  f  are  unavailable.  The  E-M  algorithm 

estimates  these  frequencies  at  each  iteration,  by  estimating  the  missing 
data  from  the  estimates  of  a,  8,  Y  of  the  previous  iteration.  Thus 


if  a(P>  ,  e(p)  and  Y(p) 
after  the  p-th  iteration, 
iteration  are 


denote  the  estimates  of  a,  £  and  y  obtained 
then  the  estimates  of  f . .  at  the  (p+l)st 


(5.4) 


£oS+1>  ■  foo  +  *0.  “<p)/<“<p)  +  s(p)) 

+  g.0  a(p^/(a^p)  +  Y^)  +  h  a('p')  , 

foP+1)  ■  f0i  +  80-  s<p)/<a<p)  +  e<p)) 

+  g.  8^PV(l-o^p)  -  Y^P^ )  +  h 


and 


f(p+l) 

10 


f  10  +  sl*  Y  ^1 


(p)/(l-a(p)  -  e(p)) 


+  g.0  Y^P^  / (a^p^  +  Y^P^ )  +  h  Y^p) 
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The  estimates  of  a,  3  and  y  are  changed  then  to  , 

g(p+l)  an(j  y(p+l)  ^  by  substituting  for  (i,j=0,l)  in 

(5.3).  The  initial  estimates  in  this  process  are 

(5.5)  f£?}  =  (f±j  +  .5) / (M  +  2)  ,  i, j=0,l  . 

These  initial  estimates  will  assure  that  at  each  iteration  , 

3^^  and  y^^  are  positive  and  f^p"^  are  alwaYs  determinable. 

Notice  that  formulae  (5.4)  provide  estimates  of  the  conditional 
* 

expectation  of  f..  given  the  sufficient  statistic  (f  ,  gn  ,  g  # ,  g  ,g  ) 

vJ #  1  0*1 

This  is  the  E  (expectation)  phase  of  the  algorithm. 

Substitution  of  in  (5.3)  provides  the  M-phase  estimates 

of  0  .  The  following  numerical  example  illustrates  this  algorithm  and 
its  convergence. 

Table  5.2 

Frequency  Distribution 
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Table  5.3 
Iterative  PMLE 


p 

a(p) 

e(p) 

Y<p> 

0 

.3942 

.2019 

.1058 

1 

.3896 

.1963 

.1053 

2 

.3894 

.1947 

.1048 

3 

.3896 

.1940 

.1045 

4 

.3897 

.1940 

.1045 

5 

.3897 

.1940 

.1045 

Convergence  to  within  e  <  5  x  10  was  attained  in  five  iterations. 

5.1.3  The  Asymptotic  Variance-Covariance  Matrix 
of  the  PMLE 

Let  =  g^  +  g1.  and  =  g^  +  g^  .  These  are  the 

number  of  observations  on  one  variable  in  which  the  values  of  the  other 
variables  are  missing.  The  score  functions  corresponding  to  (5.1)  are 
defined  as 

(5.6)  S1  = 


9£(q,B,V) 

3a 

^00 

a 


11 


g 


0- 


M, 


l-a-8-Y 
M  , 


(a+8) (l-a-6)  1-a-B 


(a+y) (1-a-Y)  1-a-Y 
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(5.7) 


=  9£(a, 8»Y) 

2  96 


11 


8. 


+ 


0* 


M, 


l-a-6-Y  (a+6)  (l-a-6)  l-a-6 


and 


(5.8) 


S3  = 


9£(g,6,Y) 

9y 


'10 


'll 


Y 


g 


0 


M 


l-a-g-y 


(a+y) (1-a-y)  1-a-y 


Finally,  since  the  family  of  density  functions  of  the  minimal 
sufficient  statistic  is  smooth  in  every  open  neighborhood  within  0 
the  Fisher  information  matrix  (FIM)  of  (a,£,y)  is  given  by  the  variance- 


covariance  matrix  of  S  =  (S^,  S^) 


Or 


yield  the  FIM, 


(5.9) 


where 


(11(6))  =  M 


A+B 

A+C 


Straighforward  manipulations 


A  +  B+C  A+B 

a 


1-a-Y 


6 

A 


+  B 


A+C 


A  1~a~g  +  C 

Y 


(5.10) 


A  = 


B  = 


C  = 


l-a-6-Y 

M, 


M(a+6) (l-a-6) 

_^1 _ 

M(a+y) (1-a-Y) 


A 

Thus,  the  asymptotic  distribution  of  the  PLME,  0  ,  is  normal  with  mean 

'Xi 


0  and  variance-covariance  matrix 


% 


-±-  (V (0 ) )  ,  where  (V(0))  =  (11(0)) 

M  rv,  ^  O, 


-1 
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Explicit  expressions  for  the  elements  of  (V(0))  are: 


% 


vn<«  '  ofe 
V12«>  ■  5ST 
<5-U)  V13<«  -  dSF 
V22«>  *  DSJ 

V23«>  ' 


V33<«  '  DJ? 


£<A  1-°~Y  +  B)  •  (A  1-”~E  +  C)  -  A2  1  , 

£(A+B)  •  (A  +  C)  -  A(A+C)  1 

|a(A+B)  -  (A+C)  •  (A  1-g~Y+  B)  J 

[(A  -+~Y  +  B+C)  (A  +  C)  -  (A+C)2  j  , 

[<A  ~ f7Y  +  B+C)A  "  <A+B>  '  W«)J 

[<A  +  B+C)  <A  +  »  -  (A+B)2]  , 


and 


(5.12) 

Det  =  ("  A  +  B+C  1 

.  r 

L  a  J 

L 

-  A 


J  -  (A+B)  •  £ (A+B)  •  (A  — g-  P  +  C)  - 

[• 


■] 


A (A+C)  +  (A+C)  •  A (A+B)  - 


(A  — |-Y  +  B)  (A+C) 


] 


PMLE  of  the  matrix  (V(6))  is  obtained  by  substituting  the  PMLE's  of 
a,  g,  y  in  formulae  (5.11)  and  (5.12). 


In  Table  5.5  we  present  numerical  values  of  the  PMLE  corresponding 
to  the  frequency  distribution  of  Table  5.4. 
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Table  5.4 

Frequency  Distribution  of  (Z^,  J.^)  and  (Z2> 


Table  5.5 

PMLE  of  (a,8>Y)  and  of  V(0)  corresponding 
to  Data  of  Tabled. 4 


0 

Of 

a 

6 

Y 

.0977 

.1122 

.1456 

.0013556 

-.0002083 

-.0003225 

V  (6) 

• 

.0015880 

-.0002094 

a- 

• 

• 

.0019071 
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5.2  Combined  PMLE  of  ECTS 

As  in  the  previous  section,  consider  any  two  requirements  in  a 

given  task,  and  let  (f  ,  g  ,  M  ,  g  ,  M  )  be  the  frequency 

%  U*  1*  *U  -I 

distribution  of  the  response  (the  sufficient  statistic)  and  let 
9  =  (&,3,Y)  be  the  cell  probabilities.  Let  =  l-a-3 

be  the  probability  of  {J^n^=  1}  and  Jl^  =  the  probability 

of  =1}  .  ni#  and  II  represent  11^^.  and  , 

respectively. 

The  unbiased  estimator  of  II..,  ,  given  by  (4.1),  can  be 

1JK. 

represented  in  the  present  framework  by  the  formula 


(5.13) 


J  f  10  +  fll  +  Ml-  -  80- 

ill*  M  +  M„ 


The  variance  of  this  estimator  is 


(5.14) 


% 


V  {nx<}  =  (a-H3)(l-a-(3)/(M+M1#) 


Similarly, 


(5.15) 


% 


V  {n  }  =  (a+y )  (1-a-y)  /  (M+M  ) 
•  1  •  1 


(5.16) 


The  PMLE  are,  on  the  other  hand. 


II  =  l-a-3 
1* 


II  =  l-a-y 


having  asymptotic  variances  of 

(5.17)  AV  (V  -  -i-  [Vu(e)  1  v22(9>  1  2  v12(e)  ] 


a, 


AV  (n-i>  --r  IT11<0>  +T33<8)  +  2  vi3(e>l  • 
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where  V  (0)  are  specified  in  (5.11).  To  illustrate  the  difference 

1J  'Xj  " 

between  the  unbiased  and  the  PMLE,  we  compare  the  asymptotic  variances 

°f  ^1*  ^*1^  and  >  with  the  estimates  of  the  parameters 

given  in  Table  5.5. 


Table  5.6 

Asymptotic  Variances  of  Estimates  of  IL  (II  „ ) 

1  •  •  1 

and  their  relative  efficiency 


Parameter 

PMLE 

Unbiased 

ARE 

ni- 

.002527 

.002551 

.990 

r-H 

.002618 

.002832 

.924 

The  asymptotic  relative  efficiency  (ARE)  of  the  unbiased  estimator, 
H^.  >  compared  to  the  PMLE,  11^  ,  is  defined  as  the  Pitman  efficiency 

(5.18)  ARE  =  AV  {ni#}  /AV  {n  }  . 

^  /S 

In  Table  5.6  we  see  that  the  ARE  of  11^  compared  to  11^  is  99%  and 

that  of  n  is  92%.  This  table  indicates  that  due  to  the  dependence 

between  different  requirements  the  PMLE  might  be  slightly  more 
efficient  than  the  unbiased  estimator. 

Returning  to  the  structure  of  the  data  set,  consider  task  (i.j). 

A 

There  are  r.  .  requirements.  Let  II.  ..  ,0N  be  the  PMLE  of  II  , 
ij  ijk(£)  ijk 

obtained  from  the  joint  frequency  distribution  of  the  k-th  and  the  £-th 

requirement  in  this  task  (k^£)  .  Let  L_  =  r  -  1  .  Since  there 

are  L  different  such  joint  distributions,  we  consider  a  weighted 
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average  of  the  L..  different  PMLE's  of  II..,  as  an  estimator. 
6  13  ljk 

Accordingly,  we  define  the  estimator 

„  L..  L 

<5-19>  V  '  ji  (kl,l)  Vm'  j!  Mu  <k’«  • 


The  variance  of  II.  .,  is 
ljk 


(5.20)  v  {n. .,  }  = 


ijk  /  L. . 
'  ij 


(  Li  j  2 

Mij(k’£)  V  {nijka): 


Z  M..(k,£) 
1=1  13 


+  2Z  £  Mi:j(k,Ji)  M..(k,r)  cov  (nijkU),  n..k(r))l 

jC  ^  v 


In  order  to  compute  the  asymptotic  covariance  between 

A 

and  II.  .,  one  needs  to  extend  the  theoretical  framework  of  the  PMLE, 

ijk(£  ) 

/S  A 

to  that  of  a  simultaneous  estimation  of  anc*  ^ijk(£0  ' 

upper  bound  for  (5.20)  can  be  obtained  by  employing  the  inequality 


(5.21)  |  cov  (n1:jk(£)  ♦  I1ijk(£^)'>l  -  (V  ^nijk(£)^  V*nijk(0^ 


1/2 


This  inequality  yields  the  following  upper  bound  on  the  asymptotic 


variance  of  II  „k  ,  namely: 


(5.22) 


AV  {IT  .,  }  < 


ijk  -  /L.  . 

‘  13 


(Lu  2 
j  z  Mii  <k’ 

2  v £=1  3 


£)  AV{IIijk(£)} 


Z  M..(k,£) 


£=1 


ij 


+  Vk’r)  (av  {filjkW)>  av  {Sljk(r)» 

Aj  ^  Aj 


1/2 
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L.  . 

j'  (AV  ,Sijk(M» 

— 

1J 

Z  M..(k,£) 

1=1  1J 


A 

n_k  will  be  called  the  combined  pairwise  MLE  (CPMLE)  of  II. 


(5.23) 


iJ 

The  CPMLE  of  the  ECTS,  X..  =  Z  w  ,  II  ,  is 

1J  k=l  i;jk  ijk 

r .  . 

^  11  ^ 

/N  J  /\ 


x. .  =  z  w. ..  n. 
iJ  k=1  ijk  ijk 


An  upper  bound  for  the  variance  of  T  is 

ij 


(5.24) 


Var  {x..}<  Zj  w2., 
1J  '  k=l  ljk 


v  {n.  }  + 

ijk 


2 L  L  w.  w. ^  |  v*  {n.  }  v*  {n 

k  <  k'  l3k  ijk  L  ijk 


ijk'}] 


1/2 


1J  .  *  to 

z  w. ..  se*  (n. })2 


k=l 


ijk 


ijk 


where  V*  {II  }  is  given  by  the  RHS  of  (5.22),  and  SE*{n.  }  is  the 


ijk 


* 


ijkJ 


square  root  of  V  {IL (an  upper  bound  for  the  standard-error)  . 

The  RHS  of  (5.24)  is  denoted  by  .  In  Section  6  we  provide 

numerical  results,  based^on  the  actual  data,  which  illustrate  the 

/\ 

effectiveness  of  using  IL  ^ 


5 . 3  CPMLE  of  Variances  and  PMLE  of  Covariances 


In  the  present  section  we  discuss  the  estimation  of  the  variances 

^  2 

and  covariances  of  X_  ,  namely  0_  and  C± (j ,  j  ^)  .  According  to  (3.5) 

2 

a  CPMLE  estimator  of  a. .  is 

ij 


28  - 


T-481 


(5.26) 


o .  .  =wT.  (V..)  w.  . 
13  -VLJ  1J 


where  the  elements  of  (V  . )  are 

ij 


(5.27) 


o.^k.k")  =• 


n. ..  (i-n. ) 

ijk  xjk 


P.  .  (k,k^)  -  n.  .  II.  .,  . 

IJ  Ijk(k  )  ijk  (k) 


,  k=k" 
♦  k=fk^ 


where 


P. . (k,k") 


is  the  PMLE  of 


P±j  (k,k") 


and 


nijk(k")’  nijk"(k) 


are  the  corresponding  PMLE  of  II..,  and  II.  .  >  ,  respectively. 

1 J  K-  1  j  k 

Thus,  the  PMLE  of  the  C  (j.j")  is 


(5.28) 


C.(j,j')  =  w' .  (K.  ..>•)  w... 
1  I'JJ  iJ 

r\f 


where  the  elements  of  the  covariance  matrix  K.  . are 

i-JJ 

1 


(5.29) 


c. . .  .-(k,£)  =  p.  ...(k  ,£)  -  n... 

1  J]  1-JJ  ijk(£)  lj  £(k) 


Finally,  the  correlations  between  complete  task  scores  are  estimated 
by  the  formula 


(5.30) 


c±(j,j 


5.4  PMLE  Adjustment  of  Task  Scores 

'Wn) 

The  complete  task  score  X;.  of  the  n-th  evaluation  is 

unavailable  if  some  of  the  requirements  of  that  task  have  not  been 
observed.  One  can,  however,  estimate  the  scores  of  the  non- available 
requirements  on  the  basis  of  the  scores  of  the  available  requirements 
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and  the  estimates  of  the  P  (k,£)  in  that  population.  The  CTS  of 
task  i.j  at  the  n-th  evaluation  is 


(5.31) 


ij 


ij 

Z 

k=l 


W  Z + 
ljk  ljk  ljk 


ij 

Z 

k=l 


(1-Z 


(n)s  T(n) 
ijk;  Jijk 


a. 


An  optimal  predictor  of  ,  for  the  square-error  loss  function, 

is  the  conditional  expectation  of  X^ 1  given  the  data,  i.e., 


(5.32)  yfn)  =  Z  W...  ?  W  (1-Z(n))  E  {j(n)l  J] 

XJ  k=1  ijk  ijk  ljk  lr_1  ijk  ^  iik'  iik'-'®'1  ’ 


k=l 


ijk 


ijk1 


where  E  is  the  conditional  expectation  of  given  the 

sample  data  This  conditional  expectation  cannot  be  determined, 

since  it  requires  the  knowledge  of  all  the  possible  joint  probabilities 

tt )  .  One  can  determine,  however,  estimates  of  the 
Jt3Yt 

3 

^  /  \ 

conditional  expectations  of  *  given  particular  observed  require¬ 

ments  by  applying  the  PMLE  of  P^  (k,i,)  and  P_^>(k,£^)  .  Weighted 

averages  of  such  estimates  can  be  considered,  with  weights  which  are 
inversely  proportional  to  the  asymptotic  variances  of  these  estimates. 


r(n) 


Let  denote  the  set  of  indexes  of  all  available  requirements 

of  task  i.j  at  the  n-th  evaluation.  Let  E. ( J )  denote  the 

ijk  ij£ 

conditional  expectation  of  J ,  given  J  ,  where  l  e  . 

ijk  &  ij£  ij 

The  PMLE  of  E.  .,  (jf“L  is 
ljk.  ij36 


(5.33) 


v  <jS>  ■  jS  + 
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in  which  IT.  and  TT..,,,,,.  are  the  PMLE  of  TT.  and  of  TT...  , 

ijk(£)  lj^(k)  ljk 

respectively,  obtained  from  the  sample  data  of  requirements  i.j.k  and 

i. j 

As  in  Section  5.1.1,  let  a  =  P[j£^  =  0  ,  =  0]  , 

3  =  P[j£^  =  0  ,  =1]  and  y  =  P[J^  =  1  ,  =  0]  .  Accordingly, 

P^k,*)  =  l-a-3-Y  and  tt  =  l-a-3  ,  ^ijjKk)  =  1-a_Y  *  Thus’ 


(5.34) 


E.  (jfn)J  =  J(n)9  (1- 
13k  ij£  ijJI 


3 


:) 


l-a-y 


a  +  y 


Furthermore,  let  (a,B,Y)  -  J<">  (1-  j^)  +  ^  • 

/n  (n)  (n) 

The  asymptotic  conditional  variance  of  E..,  (J;')  ,  given  J..0  ,  is, 

approximately,  given  by 


(5.35) 

AV{E.  (jfn)J  |j?n)?}  ss 
ljk  lji,  1  ijZ 

where 

(5.36) 

/y(n)  x-  ,  8 

Xijk(il);  ^  8  a 

T(n) 

±n 

a 

,(n) 


:S(«)'(v(g»  <’$«>> 


3  p(n)  ^  >  J  F (n)  } 


8  a  ijk(£)  *  8  3  ijk(£)  *  8  y  ijk(£)' 


JL 


-Y)2  (a+Y)2  (l-a~Y)  (l-a~Y)2 


(a+Y)4 


)-( 


(a+Y)  ‘ 


,  0  , 


-  a 


(a+Y)' 


) 
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Since  the  N  different  evaluations  are  independent,  the  dependence  between 


AAA 


.(n) 


the  MLEfs  a,  $,  Y  and  a  particular  value  becomes  negligible  when 

N  grows.  Formulae  (5.35)  -  (5.36)  are  valid  for  the  case  of 

1  j  A / 

AAA 

independent  of  (oc,$,Y)  . 


a  (n  ) 

Finally,  let  denote  the  asymptotic  variance  of  the  PMLE 

of  (5.35).  A  predictor  of  ,  based  on  the  values  of  the  available 

ij  k 

requirements  in  task  (i.j)  is  the  weighted  average 


(5.37) 


1 

T(n) 

ijk 


£  e  A 


(n) 

ij 


9  /n(n) 

ijk  ij£  ijk(£) 


where 


T(n) 

ijk 


l  £  A 


(n) 

ij 


A 


1/D 


(n) 

ijk(£) 


The  weights  in  (5.37)  are  inversely  proportional  to  the  conditional  variances 
of  the  PMLE.  A  numerical  example  of  such  prediction  of  missing  observations 
is  given  in  Section  6. 

6.  Numerical  Results 


In  the  present  section  we  provide  a  few  tables  which  demonstrate 
the  theory  developed  in  the  previous  sections.  The  results  presented  here 
are  based  on  available  data  from  N=75  evaluations  of  infantry  battalions. 
For  obvious  reasons  we  do  not  present  here  the  results  explicitly  or 
completely,  but  only  f ragmentally .  In  Table  6.1  we  provide  estimates 
of  the  probabilities  of  satisfying  requirements,  corresponding  to  four 


tasks  labeled  a,  b,  c,  and  d.  The  five  rows  corresponding  to  each  case 
present,  respectively,  the  unbiased  estimators,  the  CPMLE,  the  number 


of  applicable  evaluations  for  the  requirement 


<MSk> 


the  estimated 


^  l/2 

standard  error  (SE)  of  the  unbiased  estimator  [a„  (k,k)/M„  (k,k)  ]  , 

and  the  upper  bound  for  the  SE  of  the  CPMLE,  as  obtained  from  formula  (5.22). 
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As  seen  in  the  table,  even  in  a  case  with  a  considerable  number  of  missing 
values,  the  unbiased  estimators  and  the  CPMLE  mostly  yield  identical  results. 
The  SETs  are  also  very  close.  As  shown  earlier,  the  CPMLE  might  be  some¬ 
what  more  precise  than  the  unbiased  estimators,  but  the  difference  is  not 
consequential.  The  values  in  Table  6.1  are  estimated  upper  bounds  for 
the  SE’s  of  the  CPMLE. 

In  Table  6.2  we  present  the  estimates  of  the  ECTS  and  estimates  of 
their  SETs.  Both  the  unbiased  and  the  CPMLE  are  presented.  Again,  the 
values  obtained  by  both  estimators  are  very  close.  The  values  in  the  SE 
column  of  the  CPMLE  are  the  upper  bounds  as  in  formulae  (5.24).  These 
upper  bounds  are  of  no  value.  The  precision  of  the  CPMLE  should  be 
somewhat  better  than  that  of  the  unbiased  estimator. 

In  Table  6.3  we  provide  estimates  of  the  intertask  correlations. 

These  are  the  MLE's.  The  table  consists  only  of  the  twelve  tasks  in 
MPS  1.  It  is  interesting  to  notice  that  all  the  negative  correlations 
are  small  in  magnitude.  Based  on  an  ordinary  t-test,  the  negative 
correlations  are  not  significantly  different  from  zero.  On  the  other 
hand,  some  tasks  show  considerable  positive  correlations.  This  result 
is  intuitively  expected. 

.  Finally,  in  Table  6.4  we  present  predictors  of  missing  requirement 
scores  in  one  task,  as- computed  by  formula  (5.37).  Each  row  in  this  table 
corresponds  to  a  different  evaluation.  The  values  1.00  or  0.0  are  the 
actual  scores  of  applicable  requirements.  Values  between  0  and  1  are 
estimates  of  the  conditional  probability  of  satisfying  a  non-applicable 
(missing)  requirement,  given  the  scores  of  the  applicable  requirements 
in  that  evaluation.  A  row  consisting  of  9Ts  indicates  that  the  whole 
task  was  not  applicable.  The  above  examples  illustrate  that  the  methods 
developed  in  the  present  paper  can  be  easily  implemented  on  a  large 
size  computer. 
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Table  6.1 


TT 


ijk 


Unbiased  Estimators  and  CPMLE  of 
,  and  Their  Estimated  Standard  Errors 


Requirements 
4  5  6 


Tasks 

1 

a  0.878 
0.878 

74 
0.038 
0.038 

b  0.507 
0.507 

75 
0.058 
0.058 

c  0.887 
0.887 
53 
0.044 
0.044 

d  0.865 
0.865 
74 
0.040 
0.040 


0.587 

0.919 

0.587 

0.919 

75 

74 

0.057 

0.032 

0.057 

0.032 

0.773 

0.733 

0.773 

0.733 

75 

75 

0.048 

0.051 

0.048 

0.051 

0.959 

0.840 

0.959 

0.840 

49 

50 

0.028 

0.052 

0.028 

0.052 

0.960 

0.959 

0.960 

0.959 

75 

74 

0.023 

0.023 

0.023 

0.023 

5 

0.730  0.973 

0.730  0.973 

74  75 

0.052  0.019 

0.052  0.019 

0.932  0.947 

0.929  0.947 

74  75 

0.029  0.026 

0.030  0.026 

0.980  0.960 

0.980  0.960 

50  50 

0.020  0.028 
0.020  0.028 
0.753  0.919 

0.754  0.919 

73  74 

0.050  0.032 

0.050  0.032 


0.880  0.733 

0.880  0.733 

75  75 

0.038  0.051 

0.038  0.051 


0.959  0.833 

0.960  0.833 

49  48 

0.028  0.054 

0.028  0.054 

0.987  0.878 

0.987  0.878 

75  74 

0.013  0.038 

0.013  0.038 


8  9 

0.907  0.747 

0.907  0.747 

75  75 

0.034  0.050 

0.034  0.050 


0.857  0.979 

0.839  0.979 

21  48 

0.076  0.021 

0.078  0.021 

0.987 
0.987 
75 
0.013 
0.013 
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Table  6.2 

Unbiased  and  CPMLE  Estimates  of 
The  ECTS  and  their  Estimated  SE 


Task 

Estimators 

SE 

Unbiased 

CPMLE 

Unbiased 

CPMLE 

a 

0.8146 

0.8146 

0.0198 

0.0417 

b 

0.7489 

0.7485 

0.0281 

0.0449 

c 

0.9255 

0.9253 

0.0180 

0.0343 

d 

0.9187 

0.9188 

0.0150 

0.0280 
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Table  6.3 

MLE  of  the  Inter-task  Correlations 
in  MPS  1 


1 

2 

3 

4 

5 

6 

1 

1.00 

0.47 

0.50 

0.52 

0.32 

0.13 

2 

0.47 

1.00 

0.39 

0.33 

0.17 

0.21 

3 

0.50 

0.39 

1.00 

0.51 

0.42 

0.27 

4 

0.52 

0.33 

0.51 

1.00 

0.40 

0.44 

5 

0.32 

0.17 

0.42 

0.40 

1.00 

0.67 

6 

0.13 

0.21 

0.27 

0.44 

0.67 

1.00 

7 

0.05 

0.25 

0.31 

0.38 

0.66 

0.65 

8 

0.29 

0.02 

0.06 

0.06 

0.30 

0.06 

9 

0.41 

0.11 

0.19 

0.37 

0.38 

0.19 

10 

0.32 

0.04 

0.19 

0.53 

0.18 

0.13 

11 

-0.06 

0.08 

-0.19 

0.05 

0.21 

0.19 

12 

0.37 

0.22 

0.19 

0.27 

0.35 

0.42 

7 

8 

9 

10 

11 

12 

0.05 

0.29 

0.41 

0.32 

-0.06 

0.37 

0.25 

0.02 

0.11 

0.04 

0.08 

0.22 

0.31 

0.06 

0.19 

0.19 

-0.19 

0.19 

0.38 

0.06 

0.37 

0.53 

0.05 

0.27 

0.66 

0.30 

0.38 

0.18 

0.21 

0.35 

0.65 

0.06 

0.19 

0.13 

0.19 

0.42 

1.00 

-0.08 

0.28 

0.23 

0.25 

0.13 

-0.08 

1.00 

0.21 

-0.02 

0.07 

0.07 

0.28 

0.21 

1.00 

0.21 

-0.02 

0.08 

0.23 

-0.02 

0.21 

1.00 

-0.15 

0.29 

0.25 

0.07 

-0.02 

-0.15 

1.00 

0.21 

0.13 

0.07 

0.08 

0.29 

0.21 

1.00 
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Table  6,4 

Observed  and  Predicted  Requirement 
Scores  in  One  Task 


1 

.00 

1 

.00 

1 

.00 

0 

.0 

0 

.0 

0 

.0 

0 

.0 

1 

.00 

1 

.00 

0 

.0 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

0 

.0 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

0 

.93 

0 

.69 

3 

.00 

9 

.00 

9 

.00 

9 

.00 

9 

.00 

1 

.00 

1 

.00 

1 

.00 

0 

.93 

0 

.69 

9 

.00 

9 

.00 

9 

.00 

9 

.00 

9 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

9 

.00 

9 

.00 

9 

.00 

9 

.00 

9 

.00 

1 

.00 

1 

.00 

0 

.0 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

0 

.0 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

0 

.0 

1 

.00 

1 

.00 

1 

.00 

1 

.00 

0, 

.0 

1' 

.00 

1 

.00 

1 

.00 

1, 

.00 

0, 

.0 

1 

.00 

1, 

.00 

1 

.00 

1, 

.00 

0, 

.0 

1 

.00 

1, 

.00 

1 

.00 

1, 

.00 

1, 

.00 

1 

.00 

I 

.00 

1 

.00 

1, 

.00 

1, 

.00 

9 

.00 

9, 

.00 

9 

.00 

9, 

.00 

9, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

9, 

.00 

9, 

.00 

9, 

.00 

9, 

.00 

9, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.0 

0, 

.69 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

,00 

1. 

.00 

1, 

,00 

0. 

,0 

1 , 

.00 

1. 

,00 

1, 

,00 

1, 

,00 

0. 

,0 

0. 

.0 

1. 

,00 

1, 

,00 

1. 

,00 

0. 

,0 

1. 

,00 

1. 

,00 

1, 

,00 

1. 

,00 

1. 

,00 

1 . 

,00 

1. 

,00 

1, 

,00 

1. 

,00 

1. 

,00 

1, 

.00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

0, 

,0 

1. 

,00 

0. 

,0 

1. 

,00 

0. 

,0 

1 , 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

0. 

,0 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

,00 

0. 

,0 

0. 

,0 

1. 

,00 

1. 

,00 

1. 

,00 

0. 

,0 

1. 

,00 

1. 

,00 

0. 

,0 

1. 

,00 

0. 

,0 

1. 

,00 

1. 

00 

1. 

,00 

1. 

00 

1. 

00 

1. 

,00 

1. 

,00 

1. 

,00 

1. 

00 

0. 

0 

1. 

,00 

0. 

,0 

0. 

,0 

1. 

00 

1. 

00 

1. 

,00 

1. 

00 

1. 

,00 

1. 

00 

0. 

0 

1. 

,00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

,00 

1. 

00 

1. 

,00 

1. 

00 

1. 

00 

1. 

,00 

1. 

00 

1. 

,00 

0. 

0 

0. 

0 

1. 

,00 

1. 

00 

1. 

,00 

1. 

00 

1. 

00 

1. 

,00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

0. 

0 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1. 

00 

1.00 

0.0 

0.0 

1.00 

1.00 

0.0 

1.00 

1.00 

1.00 

1.00 

0.75 

0.0 

0.69 

1.00 

1.00 

1.00 

1.00 

0.0 

1.00 

1.00 

0.76 

1.00 

0.70 

0.0 

0.0 

0.75 

0.0 

0.69 

0.0 

0.0 

0.76 

0.65 

0.70 

0.88 

0.87 

9.00 

9.00 

9.00 

9.00 

9.00 

0.76 

0.65 

0.70 

0.88 

0.87 

9.00 

9.00 

9.00 

9.00 

9.00 

1.00 

1.00 

1.00 

0.0 

1.00 

9.00 

9.00 

9.00 

9.00 

9.00 

0.0 

1.00 

0.0 

1.00 

1.00 

0.0 

1.00 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.0 

0.0 

1.00 

1.00 

0.0 

1.00 

0.0 

1.00 

1.00 

0.0 

0.0 

0.0 

1.00 

0.0 

0.0 

0.0 

1.00 

1.00 

1.00 

0.0 

1.00 

1.00 

1.00 

1.00 

0.0 

0.0 

0.0 

1.00 

1.00 

0.76 

1.00 

1.00 

1.00 

1.00 

9.00 

9.00 

9.00 

9.00 

9.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

9.00 

9.00 

9.00 

9.00 

9.00 

0.0 

0.0 

0.0 

0.0 

1.00 

1.00 

0.0 

0.0 

1.00 

1.00 

0.76 

0.64 

0.69 

0.98 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.0 

0.0 

1.00 

0.0 

0.0 

1.00 

0.69 

1.00 

1.00 

0.0 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

0.71 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.0 

0.0 

1.00 

1.00 

0.76 

0.0 

1.00 

1.00 

1.00 

1.00 

0.0 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.0 

1.00 

1.00 

1.00 

1.00 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.70 

1.00 

1.00 

0.0 

0.0 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.75 

0.65 

0.70 

1.00 

1.00 

1.00 

0.0 

1.00 

1.00 

1.00 

0.0 

0.0 

0.0 

1.00 

1.00 

0.76 

1.00 

0.70 

1.00 

1.00 

1.00 

1.00 

1.00 

0.0 

0.0 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.71 

1.00 

1.00 

1.00 

1.00 

0.71 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

0.0 

1.00 
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Table  6.4 

Observed  and  Predicted  Requirement 
Scores  in  One  Task 
(Continued) 


1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

1, 

.00 

1. 

1. 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.0 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.0 

1. 

0, 

.0 

0, 

.0 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

1, 

.00 

1, 

.00 

1. 

0, 

.0 

1 , 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.0 

0, 

.0 

0. 

.0 

1, 

.00 

1. 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.78 

1, 

.00 

0, 

.70 

1, 

.00 

1. 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

0, 

.0 

1, 

.00 

0, 

.0 

1, 

,00 

1, 

.00 

0. 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.0 

1, 

.00 

1, 

.00 

0, 

.0 

1, 

.00 

1, 

.00 

1. 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

0, 

.76 

1, 

.00 

0, 

.70 

1, 

.00 

1. 

1 

.00 

1, 

.00 

0, 

.92 

1, 

.00 

1, 

.00 

0, 

.75 

0, 

.65 

0, 

.70 

0, 

.88 

0. 

1 

.00 

1, 

.00 

0, 

.0 

1, 

.00 

1, 

.00 

0, 

.75 

1, 

.00 

0. 

.70 

1, 

.00 

1. 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

0, 

.0 

0. 

.70 

1, 

.00 

1. 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

1. 

.00 

1, 

,00 

1, 

.00 

1. 

1, 

.00 

0, 

.0 

0, 

.0 

1. 

.00 

1, 

.00 

0, 

.0 

0, 

.0 

1. 

.00 

1, 

.00 

1. 

1, 

.00 

0, 

.98 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

1, 

,00 

1, 

.00 

1. 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

.00 

1, 

.00 

1. 

1. 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

1, 

.00 

1- 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1, 

.00 

1. 

,00 

1, 

.00 

0. 

1, 

.00 

0, 

.97 

0, 

.92 

0, 

.93 

0, 

.71 

0, 

.75 

0, 

.65 

0, 

.68 

0, 

.88 

0. 

00 

00 

00 

00 

00 

0 

00 

00 

88 

00 

00 

00 

00 

00 

00 

00 

0 

86 
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